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Introduction 

Teleoperation is widely expected to perform a wide variety of tasks in future space 
operations. In order to pursue the goal of a realistic sense of presence at the worksi e 
many manipulator designs, starting with Goertz (1954), have incorporated force feedback 
capabilities. As a result, many of the studies attempting to quantify the performance of 
various teleoperator designs have concentrated on the question “can force/torque feedback 
improve teleoperation performance? . 

Kueath (1972) studied the effects of a compliant manipulation arm and force feedback 
on manipulation of an inertial load. Results showed force feedback ^ad a large effect on 
task completion time and error (hitting wall) rate for the maze task Hill & Sails ury 
(1977) evaluated three master-slave teleoperators (the Ames Exoskeletal Master Slave, 
MA11 & MA23) with an instrumented task board which emphasized the peg-in-hole tas . 
Their results documented task completion time for the different arms as a function o peg 
tolerance and showed improved performance when force feedback was present and reported 
bare-handed operator performance. 

Recent evaluation studies (Draper et.al. 1987) have put emphasis on broadening 
the base of measurements against which task performance can be judged. Besides task 
completion time, useful measures included number of task errors, peak force and variance m 
force (see also Hannaford 1987). Although ANOVA did not show a significant completion 
time improvement due to force feedback, the other measures did indicating that force 
feedback allowed the task to be performed with higher quality if not at a faster rate. 

The JPL teleoperation Laboratory has recently developed a unique telemampulation 
system featuring advanced modes of force feedback and shared contTO }^ ed ^ h ^ S J stem . 
is described in detail elsewhere in this volume (Szakalay ,Kim, Bejczy 1989). The Enhance 
6- Axis Breadboard (ESAB) teleoperation system consists of the JPL-Stanford Force Re- 
flecting Hand Controller (recently refurbished and upgraded from the original [Bejczy & 
Salisbury, 1981] design), Puma 560 manipulator, and JPL Puma Smart Hand (Fiorini, 
1988). The master side of the system is installed in a separate control station without a 
direct view of the robot work area. Three television cameras provide top, upper left rear, 
and right rear views of the task board area. The two rear view cameras could be remotely 
controlled for focus and zoom but fixed views were used for all of the experimental tasks. 
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The ESAB system can be configured by the user in a wide variety of ways. All control 
modes and gams can be independently selected for each task space axis. Motion control 
modes include position control, rate control, and “disabled”. The system thus provides a 
rich set of control possibilities. In this study, five control modes plus direct human task 
performance were experimentally tested. However, only preliminary experiments were per- 
formed with the two modes of shared control. Because of space limitations, this paper only 

nSSS* 8 jT C ° l ntr ° 1 m ° des: position contro1 °nly, position control with force feedback 
frrij), and barehanded operator manipulation. 

This study is part of a longer term effort to quantitatively evaluate a snapshot of 
present telemampulation technology to expose improvements needed for real-world appli- 
cations. The overall approach was to design a preliminary experiment which looked at a 
relatively large number of independent variables. 

Experiment Design 

The experimental design varied control mode, task, and subject. The dependent 
measures were task completion time, sum of squared forces, and number of errors. Three 
repetitions of each sub-task were performed by each subject in each of the control modes 
All sub-tasks were performed in random sequence to form one repetition (randomization 

without replacement). All subjects performed all repetitions of a given control mode before 
the next mode was tested. 

The tasks used in experimental teleoperator evaluation fall naturally into two classes: 
generic tasks and application tasks. Generic tasks are idealized simplified tasks which 
are designed to test specific telemanipulation capabilities. Application tasks are designed 
as much as possible to mimic real world uses for teleoperation. Evaluation based on 
generic tasks illustrates the telerobotics technology push, while application tasks guide the 
technology in the direction of greatest payoff. 

, ,The task board consists of a 21” by 21” frame which accepts modules of either 7” 
x 7 or 14” x 7”. An advantage of the modular task design is that the tasks can be 
mounted individually on a six-axis force-torque sensor to enable force torque recordings 
during direct manual operation of the tasks. The four tasks used were: velcro, peg in hole 
matrix, electrical connectors, and bayonet connector. Each task is in turn broken down 
into component subtasks: 

Task 1 Velcro attachment. Exchange the position of two differently shaped blocks at- 
tached to the task board module with velcro. Attempt to attach the blocks 
securely while minimizing unnecessary force. 

Task 2 Peg in hole matrix, this task consists of nine holes arranged in a square matrix. 
The rows each have a progressively larger clearance, and each column has a 
different chamfer. The subtasks are to take the standard peg and insert it into 
a given hole. Peg/hole clearances ranged from 0.005” to 0.0026”. Pee diameter 
was 0.998”. 
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Task 3 Electrical Connectors. The subtasks consists of the mating and unmating o 
three standard electrical connectors: a 3 prong chassis power cord connector, 
DB25 25pin signal connector, and 1/4” telephone style plug. 

Task 4 Bayonet Connector. This task consists of unlocking, unmating, mating, and 
locking a Bendix bayonet style electrical connector (type PT06A-20-16S/16). 

Each task was performed according to a pre-specified procedure to which the subjects 
trained The task sequences were interspersed with “taps” in which the operator made 
momentary contact with a designed point on the task board either with the bare gripper, 
or with a held object. The “taps” injected distinct spikes in the force record (especially the 
X axis: normal to the task board surface, see Figure la). The tasks all began and en e 
with a tap on a designated square on the task board surface. The resulting force spikes 
provided well defined benchmarks for measurement and interpretation of the progress of 
the task by inspection of the force records alone. 

Five test operators for this experiment were chosen who would have technical back- 
ground, but not have in-depth knowledge of robotic technology. Detailed robotic knowledge 
or knowledge of the specific system itself was felt to be distracting and to not reflect forsee- 
able operator populations. Subjects recruited were graduate and undergraduate students 
who were not specialists in robotics. Each subject received 2 to 4 hours of practice on the 
apparatus. The practice sessions consisted of four, 30 minute sessions in which the task 
set was performed with and without force feedback. 

Analysis and Performance Measures 

The raw force torque data is a rich load of information which can be understood 
in terms of the task description and which can in turn be used to quantify task perfor- 
mance (Figure 1). This section will describe computations which were performed on the 
force/torque data to produce performance measures. Completion time, can be determined 
from the length of the data file containing the force torque data. 

Sum of Squared Force (SOSF) is computed by taking a nondecreasing sum of the 
square of the force or torque values. 

N 

SOSF = £ ffd t 

i 

where N = number of data samples (task time over dt), fi is the ith sample of force or 
torque, and dt is the sampling interval (0.01 sec in this experiment). SOSF is accumulated 
separately for each force and torque axis. A third way in which performance can be mea- 
sured is through an observer’s notations of the “quality” with which a task is performed. 
In our experiments, a set of “errors” was defined and explained to the test operators and 
experimenters. A test operator watched each repetition of the experiment and counted 

occurrences of each error. 

In some cases it is desirable to compare performance measures among different seg- 
ments of the same task. For example, to compare the completion time and SOSF for 
peg insertion vs. peg extraction. This was accomplished through a computer program 
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which could recognize benchmarks in the force signal and divide it in time between a set 
of segments. Returning to the peg-in-hole example (see “x axis force”, Figure 1) the data 
can be clearly divided into “translation” (the manipulator is in free motion: no contact 
forces), “taps” (sharp spikes in force), “insertion” (predominantly positive forces) and “ex- 
traction (predominantly negative forces). Both “completion time” and SOSF can thus be 
computed for each segment of the task. 

Results 


When experimental records for the peg-in-hole task performed in the several control 
modes are compared together (Figure 2), the X axis force traces tell most of the story 
because of the alignment between the task axis and the force/torque sensor’s X axis. 
Comparison of performance in the several control modes shows the reduced completion 
times and force levels achieved when capability is added to the system. 

Although fascinating in themselves, these raw data records are isolated anecdotes of 
individual task performances. To draw conclusions, the data were reduced to the three 
basic performance measures. Records of this type for each repetition of each task were 
processed to produce the performance data points upon which the results below are based 

The visually scored error rates were manually correlated with the reduced performance 
data. 


The completion time, SOSF, and number of errors data can be simplified by aver- 
aging across one or more dimensions of the design. As a first look at the data, we have 
computed averages over all subjects and over the first three tasks, ” velcro”, ”peg-in-hole” 
and "electrical connectors”. The fourth task was not included in this average because it 
took significantly longer than the others (approximately 150 seconds vs. 75 seconds), and 
was often not completed due to its difficulty. There are 9 subtasks for the peg-in-hole task 
(corresponding to the 9 test holes) vs. 2 for the velcro and 3 for the electrical connectors. 
These averages (Figures 3a, 3b, and 3c) show clear trends in performance as the level of 

capability progresses from position control, through force reflection, up to the bare handed 
operator. 


Completion time (Figure 3a) for the three primary tasks drops from an average of 92 
seconds with position control to an average of 63 seconds when force feedback is added. 
Completion time drops to only 14 seconds for the bare-handed human. SOSF (Figure 3b) 
drops even more dramatically (from about 3500 to 500 lbs 2 sec) when pure position control 
is augmented with force feedback and further (to 200 lbs 2 sec) for the bare-handed case. 

The number of errors observed (Figure 3c) per repetition drops from 3.0 to 1.1 as 
force feedback is added. No errors were observed in the bare-handed data. 


The probability of the null hypothesis that there was no effect of force feedback (cal- 
culated by the two-tailed Z test) was much less than 0.01 in all of the differences reported 
above giving them a high degree of statistical significance. 


These results summarize one of the main results of this study, that the provision of 
force feedback reduces completion time for a task mix emphasizing energetic interaction 
and precision manipulation by approximately 30%, reduces SOSF by a factor of 7, and 
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reduces errors in performing the task by 63%. 

The first level of detail to add to the summary results is to break them down by 
individual task (Figures 3d, 3e) instead of averaging all the tasks together. Doing is 
shows that the effect of force feedback is not the same for all of the tasks. For the peg-in- 
hole task, completion time (Figure 3d) follows the expected course dropping by almost a 
factor of two from 105 to 59 seconds as force reflection is added. For the velcro blocks as , 
completion time increases from 72 to 83 seconds. Both of these changes are statistically 
significant by the method described above. For the electrical connectors, only a shg 
change is observed which was NOT statistically significant. Of course all of the tasks were 
completed much faster by the bare-handed operator. The average time in this case is about 

15 seconds. 


The SOSF data (Figure 3e) tell a different story. As with completion time, for the peg- 
in-hole task there is a dramatic drop in SOSF (from 5400 to 500 lbs 2 sec) as force reflection 
is added. The increase in completion time seen for the velcro task is accompanied y a 
significant decrease in SOSF (from 800 to 400 lbs 2 sec). For the electrical connectors, the 
SOSF measure declines significantly in spite of their unchanged completion time. 


The performance measures were calculated for the segments of 150 repetitions of 
the peg-in-hole task (Figure 4). Total time (Figure 4a) spent in the movement phase is 
unchanged at 32 sec by the addition of force feedback. The tap phase is also unchanged a a 
negligible 2 sec. But the insertion and extraction phases are accomplished markedly faster 
(31 vs. 11 for insertion, 35 vs. 10 for extraction) when force feedback is present. The two 
pie charts (Figure 4b, c) illustrate the changing nature of the task mix. As force reflection 
is added the dominant component of completion time changes from insertion/extraction, 
the environmental interaction phases, to the free motion phase of the task. 


Conclusions 

This paper has presented a few of the results of a major study of over 100 hours of 
experimental teleoperation. Force and torque data recorded from the robot wrist is a rich 
source of information on the performance of tasks. Performance measures can be computed 
for whole tasks, or for specific task segments. As a general principle, the performance 
increases as manipulation capability is increased although the effects may depend on tas 
and performance measure. This study has laid the groundwork for much future work 
Further reports will detail additional results which could not be presented here due to lack 
of space as well as follow-on experiments investigating manipulation under time delay and 
shared control conditions. 
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